Moving standardised exams online won’t solve our testing problem

Covid-19 pushed the May 2020 Advanced Placement exams online, a move that led to various technology and security issues, and resulted in a lawsuit against the exam’s creator, the College Board.

Meanwhile, spring and summer dates for the SAT – the standardised test widely used in US college admissions – were cancelled and College Board CEO David Coleman has said that if high schools are closed in the fall, these traditionally in-person assessments should also be administered online.

While giving as many students as possible access to these exams is admirable and important, test providers should keep in mind that it’s simply not enough to move these paper tests online, especially during a global pandemic when demand for everything to be digital puts pressure on timescales. Moving tests online presents technological challenges and raises questions around security and accessibility.

The SAT was first taken by high school students in 1926. Although it has undergone several format and scoring changes over the years, what’s remained constant is the setting: large groups of students gathered in rooms taking the same test simultaneously.

In 2018, the College Board allowed districts in certain states the ability to offer the exam digitally. While safeguards were put in place to test the strength of a building’s network and guard against potential test theft, the exam itself remained essentially the same as the hard copy

The crux of the matter is that paper tests – or any static, fixed-form test for that matter, even if administered by computer – are designed to accommodate the testing site, rather than test takers themselves. They are fundamentally not “digital-first.”

One problem with moving paper tests online is that the environment is no longer standardised. In a traditional setting, students are given the same test forms, to be taken at the same place, at the same time, using the same equipment. However, remote testing introduces many new variables, such as different computer hardware or internet speeds. One way to mitigate the effects of these differences is to design tests to be shorter and more efficient, thus minimising technological issues that may arise regardless of the equipment used.

A second, related problem is that traditional paper tests are designed to be very long. This is because the student must be assessed by questions that cover the entire range of ability, from very basic to very advanced. As a result, the SAT can take three hours to complete – nearly four hours if a student opts for the essay portion, which is required or strongly encouraged by many elite institutions.

Test fatigue is real. After taking the then new version of the exam in 2016, many students commented that they felt exhausted afterward.

But we have better tools now, and we should use them. Test developers years ago recognised the potential of computer-adaptive tests which adjust to the abilities of the student by allowing them to respond to questions, and for the software to adjust and scale difficulty based on performance.

It has been shown that adaptive tests can be completed in a fraction of the time while maintaining uniformly precise scores. The Graduate Record Examination (GRE) and Graduate Management Admission Test (GMAT) are two examples.

A third problem is test integrity and security. One common strategy for gaming a paper test is to secure and circulate the test items in advance of a scheduled administration. Computer adaptive tests help combat this, since every test administration is tailored to each individual student. However, this only works if the item bank to draw from is large enough. Unfortunately, most modern computer-adaptive tests are still designed to be taken at test centres, where it is more challenging to copy and circulate items, so they have relatively small test item banks.

Digital-first tests can and should take advantage of artificial intelligence and machine learning to, among other things, aid in generating many thousands of test items – a feat nearly impossible to execute efficiently and consistently by more traditional test-development processes.

For example, a student would have to take the Duolingo English Test, an online adaptive test that uses AI-assisted item development and which I helped create, roughly 1,000 times before seeing the exact same item again. With students taking standardised tests across the globe in different time zones, this is a critical aspect of basic exam integrity.

A fourth and final problem with flipping a paper exam to digital during a global crisis is a lack of time to develop appropriate proctoring and security protocols. Artificial intelligence can also play an important role here: computer vision and biometrics techniques can be used in tandem with live or asynchronous remote human proctoring. Even verifying a student’s identity or locking down their browser and auxiliary applications are non-trivial, both ethically and technologically. These systems can take months or years to develop and should not be haphazardly deployed overnight.

Exams that were designed to be online, digital-first assessments provide a pedagogical experience different to traditional paper, static exams. They are, by design, flexible, consistently evolving, and empirically driven. The opportunity is here for test providers to learn to be nimble, and rethink how tests can be developed to meet the needs of the test taker over those of the test centre. That opportunity requires more research and effort than simply scanning in a paper test.

Burr Settles is a scientist and software engineer specialising in statistical machine learning and human-computer interaction. He is the research director for Duolingo and helped create the Duolingo English Test.